TerminalBench Bot Commands

Run tasks:
/bot tb run [--dataset, --dataset-path, --dataset-config, --registry-url, --local-registry-path, --output-path, --run-id, --upload-results, --task-id, --n-tasks, --exclude-task-id, --no-rebuild, --cleanup, --use-subscription, --model, --agent, --agent-import-path, --agent-kwarg, --log-level, --livestream, --n-concurrent, --n-attempts, --global-timeout-multiplier, --global-agent-timeout-sec, --global-test-timeout-sec, --contributionsCommit]

Check:
/bot tb tasks check [--task-id, --tasks-dir, --unit-test-relative-path, --dockerfile-relative-path, --model, --agent, --fix, --output-path, --contributionsCommit]

Debug:
/bot tb tasks debug [--task-id, --run-id, --runs-dir, --tb-run-job-id, --tasks-dir, --agent, --model, --n-trials, --output-path, --contributionsCommit]

Full Check:
/bot full-check-v2 [--task-id <id>] [--analyze-failure] [...]
/bot full-check-v2 --opus-only - Run only tb_run_large with Claude Opus 4.7
/bot full-check-v2 --sonnet-only - Run only tb_run_large with Claude Sonnet 4.5
/bot full-check-v2 --tbench - Oracle + NOP + tb_run_large (Codex + openai/gpt-5.5, 5 parallel attempts, subscription; requires 0-3/5 resolved) + Codex harbor debug/analyze when ≤3/5 resolved
/bot full-check-v2 --tbench --opus - Same as --tbench, but tb_run_large uses Claude Opus 4.8 via claude-code
/bot full-check-v2 --fusion-reports - Oracle + NOP + harbor_run_large (Codex + openai/gpt-5.5, 1 attempt, subscription). No fallback stage. Harbor saves /output, /app/output, and traces per trial automatically (implicit --artifacts — needed for downstream re-verify).
/bot full-check-v2 --oracle-nop-only - Lightweight gate: only Oracle (5 attempts, 5 retries) + NOP, skip every model run / similarity / debug / quality check (~30-60s per task)
/bot full-check --tb-run-large-agent claude-code --tb-run-large-model claude-opus-4-7

MM Trace Run:
/bot mm-trace-run - Oracle + NOP + 5 GPT-5.5 xhigh trials (codex, subscription) with artifacts/traces. Posts per-trial rewards and step counts, avg reward, avg steps, max/avg reward ratio, and artifact + trace viewer links.

Re-verify (re-score existing agent attempts against updated tests):
/bot re-verify [--task-id <id>] [--skip-oracle] [--skip-nop] - Re-run tasks/<task-id>/tests/test.sh against every trajectories*/<task-id>/<agent>/<N>/artifacts/output/ directory in the PR head ref (1 claude + 4 grok by convention). Also runs oracle (canonical solution/solve.sh, expected reward 1.0) and nop (empty /output, expected reward 0.0) sanity rows by default — pass --skip-oracle or --skip-nop to opt out. Produces fresh per-attempt rewards without re-running the agents — useful after editing tests/ during review.

Harbor format checker:
/bot harbor-format-check [--trace-s3-url s3://bucket/prefix[,s3://bucket/other-prefix]] [--policy mm-abc|compat] - Run the standardized pre-acceptance format checker on this PR, including LLM fuzzy checks and optional S3 trace checks.

For detailed parameter descriptions, run tb --help or tb <command> --help locally.

Job Management:
/bot job list - List all running jobs
/bot job status <job_id> - Get status of a specific job
/bot job kill <job_id> - Kill a running job
/bot job restart <job_id> - Restart a failed job
/bot job info <job_id> - Show detailed information about a job
/bot job cleanup - Remove all failed-to-report jobs

Review:
/bot review - Trigger AI code review on this PR

Remove default flags: Use --no-{flag} to disable default flags (e.g., --no-use-subscription)

Aliases:
/bot /codex-attempts [--n, --task-id, --tasks-dir] → /bot /tb run --agent codex
/bot /claude-attempts [--n, --task-id, --tasks-dir] → /bot /tb run --agent claude-code
/bot /grok-attempts [--n, --task-id, --tasks-dir] → /bot /tb run --agent terminus-2 --model xai/grok-4.3-internal --agent-kwarg reasoning_effort=high
/bot /tb-check [--task-id, --tasks-dir] → /bot /tb tasks check
/bot /oracle [--task-id, --tasks-dir] → /bot /tb run --agent oracle
/bot /nop [--task-id, --tasks-dir] → /bot /tb run --agent nop
/bot /tb-debug [--task-id, --tasks-dir] → /bot /tb tasks debug

Get help: /help or /bot help

after having a proper think about this I think its easier if we assume the judge only has access to the video recording

If it wants screenshots it can take them from there

otherwise the judge will need to know about the concept of a grid screenshot vs a non-grid screenshot and which paths each has. to me that seems like too much dependency

for now I think lets keep the screenshots as a purely playtester-facing + debugging artifact

this also allows us to avoid a little complexity, which is not insignificant

thoughts?